Effects of Early Warning Emails on Student Performance

class: title-slide

<br>
<br>
# Effects of Early Warning Emails on Student Performance <br>

.padding_left.pull-down.white[
.bold[_J. Klenke_], T. Massing, N. Reckmann, J. Langerbein, B. Otto, M. Goedicke, C. Hanck
<br>
<br>
<br>
`$15^{TH}$` International Conference on Computer Supported Education

Prague, 21-23 April, 2023
]

---
# Research Idea and Course Description

- __Research Idea:__ students should receive objective and motivating warning email   
- Analyzed Course: _Inferential Statistics_ at the University of Duisburg-Essen of the summer semester 2019
- Compulsory course for several business and economics programs
  - Weekly 2-hour lecture 
  - Weekly 2-hour exercise
  - [Kahoot!](https://kahoot.com/) games were used to interact with students during classes
  - Homework and 5 online tests were offered on the e-assessment platform  [JACK](https://s3.paluno.uni-due.de/en/forschung/spalte1/e-learning-und-e-assessment)
- Information from __802__ individuals were collected from Moodle  
  - __337__ students took an exam at the end of the semester

---
# Data and Decision Rule
- We used two data sources 
  - First three online test results  
  - Cumulative points of the tasks in JACK
- Logit model was used to predict students probability to pass the final exam based on the frist 3 online tests
  - Model was trained with the latest data obtained from previous edition of the same course
- If predicted probability to pass `$\leq 0.4$` the student got a warning mail

---
# Course Timeline Main Events

<br>

<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#plots/timeline_plot.png" alt="Timeline for the key events in the 2019 summer term course Inferential Statistics (treatment cohort)." width="80%" />
<p class="caption">Timeline for the key events in the 2019 summer term course Inferential Statistics (treatment cohort).</p>
</div>

- The shaded area indicates the period after treatment. 
- 57 days between the warning mail and 1st exam 
- 113 days between the warning email and 2nd exam

---
# Literatur

- Purdue

-> Problem : Kaum analisiert 
---
# RDD Example Plot &#8212; I

## Parametric Estimation

.pull-right[
- Groups left and right of the cutoff 
- Local Average Treatment Effect (__LATE__)
]

---
# RDD Example Plot &#8212; II

## Non-parametric Estimation

- We employed the data-driven approach by <a id='cite-imbensoptimal2009'></a><a href='#bib-imbensoptimal2009'>Imbens and Kalyanaraman (2009)</a> to determine the bandwidth

???
- The method fits the bandwidth as widely as possible without introducing other confounding effects

---
# Model Assumptions

- The running variable `$W$` (predicted probability to pass the exam) needs to be continuous around the cutoff, otherwise students could manipulate the treatment

.pull-left-2[
<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#plots/test_cont_label.png" alt="Graphical illustration of the McCrary sorting test." width="80%" />
<p class="caption">Graphical illustration of the McCrary sorting test.</p>
</div>
]

.pull-right-1[
<br>

- There is no jump in the density around the cutoff point of `$0.4$`. 
- `$p$`-value: `$0.509$`
- The incentive to manipulate the treatment is quiet low.

]

.pull-down[
- Also standard IV estimation assumptions must hold, which they do by construction of RDD
]

---
# Empirical Results

.pull-left-2[

<br>
<div class="figure" style="text-align: center">
<img src="data:image/png;base64,#plots/model_plot_label.png" alt="Graphical illustration of the RDD model." width="80%" />
<p class="caption">Graphical illustration of the RDD model.</p>
</div>
]

.pull-right-1[
### Estimates
- LATE: 0.193 (4.889)
- Bandwidth: 0.255
- `$F$`-statics: 0.257
- `$N$` : 126

<br>
- We also estimated the RDD with covariates 
  - The results were identical
]

???

- theoretisch sollten Sie jetzt einen großen Sprung sehen
---
# Empirical Results

- The LATE estimate is positive but not significant  
  - An estimate of `$0.193$` means that students who received the warning email achieved `$0.193$` points more than comparable students who did not 
  - Compared to the `$60$`-point exam, the effect size seems limited
- The bandwidth of `$0.255$` was determined with the data-driven approach of <a href='#bib-imbensoptimal2009'>Imbens and Kalyanaraman (2009)</a>.
  - Only students with a predicted probability `$0.4$` (cutoff) `$\pm 0.255$` (bandwidth), are included in the analysis.
- This leads to the sample size of `$126$` students.

---
# Discussion

- Our RDD results do not provide evidence that the warning email has a significant effect on students’ results (or behavior)
- The variance around the cutoff is rather high, which compromises the detection of an effect 
- Many individuals are not included in the final analysis for several reasons
  - Students dropping the course
  - Students far away from the cutoff are not provding much information for the model

-> Thus precise estimation of the treatment becomes more difficult

---
# Discussion

- Students get also feedback through their online tests
- The warning might may also lead weak students to postpone participation to a later semester
- The motivation and objective feedback from one warning email is rather small

---
# Further Research

- The effect on the dropout rate from such warning emails or systems requires further attention 
- An automatic repeatedly feedback system could possibly have a greater impact on students motivation
  - A detialed recurring feedback could also used to guide students

<br>
.blockquote[
We see the open and transparent communication of the student’s performance to the students as a positive aspect of the system.
]

---
# Sources

.font90[
- Angrist, J. D., Imbens, G., and Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434):444–455. Publisher: Taylor & Francis.
- Huntington-Klein, N. (2022). The effect : an introduction to research design and causality. Chapman & Hall
book. CRC Press, Taylor & Francis Group, Boca Raton ; London ; New York, first edition edition.
- Imbens, G. and Kalyanaraman, K. (2009). Optimal bandwidth choice for the regression discontinuity estimator. National Bureau of Economic Research, 1(14726).
- McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142(2):698–714.
]

---
# References

Imbens, G. and K. Kalyanaraman (2009). "Optimal Bandwidth Choice for
the Regression Discontinuity Estimator". In: _National Bureau of
Economic Research_ 1.14726.

---
# Appendix

---
# Regression Discontinuity Design (RDD)

- The treatment is __not__ randomly assigned and therefore methods like OLS are not suitable
  - Treatment is a function of the predicted probability to pass the exam   
- Consider the following _sharp_ RDD representation <a id='cite-Huntington-Klein2022'></a><a href='#bib-Huntington-Klein2022'>Huntington-Klein (2022)</a>:

`$$Y_i = \beta_0 + \alpha T_i + \beta W_i + u_i$$`
.padding_left.padding_left.font80[
<ul>
  <li> $W_i$ denotes the predicted probability to pass the final exam</li>
  <li> $T_i$ indicates if a student received a mail</li>
  <ul style="list-style-type: '&#8227;  ';"  >
    <li>$T_i = 1[W_i \leq c]$ , with $c = 0.4$ </li>
  </ul>
  <li>$\alpha$ denotes the treatment effect</li>
  <li>$u_i$ denotes the error term</li>
</ul>
]

---
# Fuzzy RDD

- __Fuzzy__ RDD allows to analyse a treatment in a setting where the two groups are not perfectly separated
  - Only the likelihood of receiving the treatment needs to _change_
- The effect is estimated through an instrumental variable estimation 
<a id='cite-angristidentification1996'></a><a href='#bib-angristidentification1996'>Angrist, Imbens, and Rubin (1996)</a> where in the first stage the `$\widehat{T}_i$` are estimated which then are inserted in the second stage

- First Stage: 
`$$T_i = \gamma_0 +\gamma_i Z_i + \gamma_2  W_i + \nu_i \qquad \quad$$`

- Second Stage: 
`$$Y_i  = \beta_0 + \alpha \widehat{T}_i + \delta_1 W_i + \beta X_i + u_i$$`

---
# Regression Discontinuity Design (RDD) II

.font70[
<ul>
  <li>RDD compares the individuals around the cutoff to estimate the effect</li>
  <li><strong>Main Assumption:</strong> Individuals around the cutoff are compareable and only differ in the treatment assignment</li>
  <ul>
    <li>The estimate is called Local Average Treatment Effect (<strong>LATE</strong>) </li>
  </ul>
  <li>Two estimation approaches: parametric and non-parametric</li>
  <ul>
    <li>Parametric estimation</li>
    <ul>
      <li>Uses the whole sample size but (many) more parameters</li>
      <li>Individuals away from the cutoff are less relevant for the estimation of the effect</li>
    </ul>
    <li>Non-parametric estimation</li>
    <ul>
      <li>Only <em>comparable</em> (near the cutoff) individuals are used for the analysis</li>
      <ul>
        <li>Decision whether to include an individual depends on the running variable $W$</li>
        <li>The groups are determined by the the data-driven approach of Imbens and Kalyanaraman (2009)</li>
        <ul>
          <li>With an $F$-test the bandwidth is determined</li>
          <li>$c \pm bandwidth$ are the two groups</li>
        </ul>
      </ul>
    </ul>
  </ul>
</ul>
]